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Abstract 

Formal proofs provide detailed justification for the validity of 
claims and are widely used in formal software development 
methods. However, they are often complex and difficult to 
understand, because the formalism in which they are con- 
structed and encoded is usually machine-oriented, and they 
may also be based on assumptions that are not justified. This 
causes concerns about the trustworthiness of using formal 
proofs as arguments in safety-critical applications. Here, we 
present an approach to develop safety cases that correspond to 
formal proofs found by automated theorem provers and reveal 
the underlying argumentation structure and top-level assump- 
tions. We concentrate on natural deduction style proofs, 
which are closer to human reasoning than resolution proofs, 
and show how to construct the safety cases by covering the 
natural deduction proof tree with corresponding safety case 
fragments. We also abstract away logical book-keeping steps, 
which reduces the size of the constructed safety cases. We 
show how the approach can be applied to the proofs found by 
the Muscadet prover. 

1 Introduction 

Demonstrating the safety of large and complex software- 
intensive systems requires marshalling large amounts of di- 
verse information, including models, code, specifications, 
mathematical equations and formulas, and tables of engineer- 
ing constants. Obviously tools supported by automated analy- 
ses are needed to tackle this problem. For the highest 
assurance levels, these tools need to produce a traceable 
safety argument that shows in particular where the code and 
the argument itself depend on any external assumptions. 

However, many tools commonly applied to ensure software 
safety rely on techniques such as static analysis [20] or model 
checking [7] that do not produce enough usable evidence (i.e., 
justification for the validity of their claims) and can thus not 
provide any further insights or arguments. In contrast, in for- 
mal software safety certification [8], as in other formal soft- 
ware development methods [3, 5], formal proofs are available 
as evidence. However, these proofs are typically constructed 


by automated theorem provers (ATPs) based on machine- 
oriented calculi such as resolution [21]. They are thus often 
too complex and too difficult to understand, because the for- 
malisms spell out too many low-level details. Moreover, the 
proofs may still be based on assumptions that are not valid, or 
may contain steps that are not justified. As consequence, con- 
cerns remain about using these proofs as arguments rather 
than just evidence in safety-critical applications. In this paper 
we address these concerns by systematically constructing 
safety cases that correspond to formal proofs found by ATPs 
and explicitly highlight the use of assumptions. 

The approach presented here combines abstraction and visu- 
alization to reveal and present the proofs underlying argu- 
mentation structure and top-level assumptions. We work with 
natural deduction (ND) style proofs, which are goal-directed 
and thus closer to human reasoning than resolution proofs, 
and we show how the approach can be applied to the proofs 
found by the Muscadet ATP [19]. We explain how to con- 
struct the safety cases by covering the ND proof tree with 
corresponding safety case fragments. The argument is built in 
the same top-down way as the proof: it starts with the original 
theorem to be proved as the top goal and follows the deduc- 
tive reasoning into subgoals, using the applied inference rules 
as strategies to derive the goals. However, we abstract away 
the ATP’s book-keeping steps, which reduce the size of the 
constructed safety cases. The safety cases thus provide a 
“structured reading guide” for the proofs that allows users to 
understand the claims without having to understand all the 
technical details of the formal proof machinery. This paper is 
a continuation of our previous work to construct safety cases 
from information collected during the formal verification of 
the code [4], but here we concentrate on the certification 
components, i.e., the domain theory and the ATP used to sup- 
port the software safety assurance process. 

2 Formal Software Safety Certification 

Our work is set in the context of formal software safety certi- 
fication [8], where we use formal source code analysis tech- 
niques based on program logics to show that the program 
does not violate certain conditions during its execution. A 
safety property is an exact characterization of these condi- 
tions, based on the operational semantics of the programming 
language. Each safety property thus describes a class of haz- 



ards. In our framework, the safety property is enforced by a 
safety policy , i.e., a set of verification rules that derived from 
initial set of safety requirements that formally represent the 
specific hazards identified by a safety engineer, and derive a 
number of logical proof obligations. Showing the safety of a 
program is thus reduced to formally showing the validity of 
these proof obligations: a program is considered safe wrt. a 
given safety property if proofs for the corresponding safety 
proof obligations can be found. Formally, this amounts to 
showing 

D U A 1= P => C (1) 

for each obligation, i.e., the formalization of the underlying 
domain theory D and a set of formal certification assumptions 
A entail a conjecture, which consists of a set of preconditions 
P that have to imply the safety condition C. The domain the- 
ory formalizes the extra-logical operations that occur in the 
obligations; it includes arithmetic functions and relations, 
programming language operations such as array indexing as 
well as application-specific operations such as matrix inver- 
sion. Assumptions typically specify global properties required 
by the component (e.g., the physical units of the input sig- 
nals), while preconditions and safety conditions refer to prop- 
erties at intermediate locations in the code. 

The different elements of these proof obligations have differ- 
ent origins, and thus different levels of trustworthiness, and a 
safety case should reflect this. The premises and the safety 
condition are inferred from the program by a trusted software 
component implementing the safety policy, and their con- 
struction can already be explained in a safety case [4]. In con- 
trast, both the domain theory and the assumptions are 
manually constructed artifacts that require particular care. The 
main hazard that we address in the safety cases here, by mak- 
ing explicit the use of hypotheses, is the unintended introduc- 
tion of logical inconsistencies that can be exploited by the 
ATPs to construct logically correct but vacuous proofs. 

The necessary analysis is hampered by the fact that the prover 
does not work on the obligations in the form as given in (1) 
but uses the form 

1= A(DuA)aP=>C (2) 

which is logically equivalent but blurs the distinction between 
domain theory, assumptions, and preconditions, and lumps 
them all together as logical premises. However, proofs typi- 
cally use only a subset of these premises as hypotheses, and 
the safety case should make explicit those that are actually 
used. In particular, it needs to highlight the use of assump- 
tions. These have been formulated in isolation by the safety 
engineer and may not necessarily be justified, and the possi- 
bility of a logical inconsistency with the domain theory is 
substantially higher. Moreover, fragments of the domain the- 
ory and the assumptions may be used in different contexts, so 
the safety case must reflect which of them are available at 
each context. By elucidating the reasoning behind the certifi- 
cation process and drawing attention to potential certification 
problems, there is less of a need to trust the certification tools, 
and in particular, the manually constructed artifacts. 


3 Converting ND Proofs into Safety Cases 

3.1 Natural Deduction 

Natural deduction [13, 16] is a form of proof that attempts to 
provide a foundational yet intuitive system to construct for- 
mal proofs. It consists of a collection of proof rules that ma- 
nipulate logical formulas and transform premises into 
conclusions. A conjecture is proven from a set of assumptions 
if a repeated application of the rules can establish it as con- 
clusion. The proof rules can be divided into basic rules, de- 
rived rules (which can be seen as proof “macros” that group 
together multiple inference steps) and replacement rules 
(which are derived rules for equivalence and equality han- 
dling). Here, we focus on some of the basic rules; a full expo- 
sition of natural deduction can be found in the literature [16]. 

Natural deduction uses two sets of rules for each logical con- 
nective or quantifier (a,v,...,V, ...), where one introduces the 
symbol, while the other eliminates it. In the introduction 
rules, the connective or quantifier is used as the top-level op- 
erator symbol of the unique conclusion, while it occurs in the 
elimination rules in the same role in one of the premises. 

3.2 Conversion Process 

Natural deduction proofs are simply trees that start with the 
conjecture to be proven as root, and have given axioms or 
assumed hypotheses at each leaf. Each non-leaf node is recur- 
sively justified by the proofs that start with its children as new 
conjectures. The edges between a node and all of its children 
correspond to the inference rule applied in this proof step. 
The proof tree structure is thus a representation of the under- 
lying argumentation structure. We can use this interpretation 
to present the proofs as safety cases [17], which are structured 
arguments as well and represent the linkage between evidence 
(i.e., the deductive reasoning of the proofs from the assump- 
tions to the derived conclusions) and claims (i.e., the original 
theorem to be proved). The general idea of the conversion 
from ND proofs to safety cases is thus fairly straightforward. 
We consider the conclusion as a goal to be met and the prem- 
ise^) as a subgoal(s); we further consider the applied infer- 
ence rule as the strategy that shows how the conclusion is 
met. For each inference rule, we define a safety case template 
that represents the same argumentation. The underlying simi- 
larity of proofs and safety cases has already been indicated in 
[ 1 7] but as far as we know, this idea has never been fully ex- 
plored or even been applied to machine-generated proofs. 

The conversion we present here preserves the inferences and 
formulas of the original proof, but avoids overloading the 
constructed arguments with trivial proof steps. We identify 
semantically related or repeated identical inferences that can 
be abstracted away in order to construct a more concise ar- 
gument. In the following we describe the safety case tem- 
plates for the base rules of the calculus. We use the Goal 
Structuring Notation [17] to explicitly represent the logical 
flow of the proofs argumentation structure. 



3.3 Conjunctions 

The rules for conjunction introduction and elimination shown 
in Figure 1 directly represent the intuitive interpretation of 
conjunctions: if A is true and B is true, then evidently AaB is 
true as well (A-i), and if AaB is true, then both A and B must 
be as well (A-el resp. A-e2). The hypotheses that are available 
to show AaB true are also available to show A (resp. B) true 
as well. Similarly in the case of A-el (resp. A-e2) where the 
available hypotheses to show each conjunct true are also 
available to show AaB true. 


sider each of the two cases for the disjunction to be true. In 
the first case we thus assume A together with the available 
hypotheses and try to derive C, in the second case we assume 
B together with the available hypotheses and try to derive C. 
If both cases succeed, we can conclude C. The safety case 
fragment makes this argument explicit, and, in particular, 
explicitly justifies the use of the respective assumptions in the 
two cases. Figure 2 shows the safety case fragments for the 
disjunction rules. 

3.5 Implications 
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Figure 1: Safety Case Templates for A-Rules. 

The A-rules can be directly converted into safety cases. In the 
case of A-introduction, the satisfaction of the conclusion (i.e., 
goal of the safety case) is implied by the satisfaction of the 
two premises (i.e., subgoals of the safety case) based on the 
strategy of A-introduction rule. For the A-elimination rule, the 
strategy shows a logically stronger goal we can conclude A 
(resp.B) if we have a proof of AaB. Figure 1 shows the safety 
case fragments for the conjunction rules. 


The implication elimination follows the standard pattern but 
in the introduction rule we again temporarily assume A as 
hypothesis together with the list of other available hypothe- 
ses, rather than deriving a proof for it. We then proceed to 
derive B, and discharge the hypothesis by the introduction of 
the implication. The hypothesis A can be used in the proof of 
B, but the conclusion A=>B no longer depends on the hy- 
pothesis A after B has been proved. 
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B 
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A A => B 
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3.4 Disjunctions 


Figure 3: Safety Case Templates for =>-Rules. 


A disjunction can be introduced as long as one of the dis- 
juncts is already known i.e., if A (resp. B) is true, then evi- 
dently AvB is true as well. In the safety case, a goal AvB is 
constructed, which is justified by the subgoal A (resp. B) via 
the strategy (v-ii) (resp. v-i 2 ). The hypotheses that are avail- 
able to show AvB true are also available to show subgoal A 
(resp. B) true. 

M [B] 


A 

AVB 


v-n 


B 

AvB 


V-h 


AVB C 
C 


C 


V-e 


In the safety case fragment (see Figure 3), we use a justifica- 
tion to record the use of the hypothesis A, and thus to make 
sure that the introduced hypotheses are tracked properly. 

3.6 Universal Quantifiers 

The natural deduction calculus can also be used for proofs in 
predicate logic. The proof rules focus on the replacement of 
the bound variables with objects and vice versa. For example, 
in the elimination rule for universal quantifiers, we can con- 
clude the validity of the formula for any chosen domain ele- 
ment t x . 



Figure 2: Safety Case Templates for v-Rules. 

In contrast, in disjunction elimination, we only know that 
AvB holds, but not which of A or B is true, so that we need to 
reason by cases to conclude C from AvB, i.e., separately con- 
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Figure 4: Safety Case Templates for V-Rules. 


In the introduction rule, however, we need to show it for an 
arbitrary but fresh object t x (that is, a domain element which 
does not appear elsewhere in H, A, or the domain theory and 
assumptions). If we can derive a proof of A, where x is re- 
placed by the object t x , we can then discharge this assumption 
by introduction of the quantifier. The safety case fragments 
(see Figure 4) record this replacement as justification. The 
hypotheses available for the subgoals in the V-rules are the 
same as those in the original goals. 

4 Safety Case Generation Process 

To automatically construct the ND proof safety case, we inte- 
grate the Muscadet [19] theorem prover with Adelard’s ASCE 
tool [1]. We convert the proofs that are generated by the Mus- 
cadet prover into an XML format (i.e., PROOF-XML in Fig- 
ure 5). The XML file contains all the relevant information 
that is required for the automatic safety case construction. 
Subsequently, an XSLT program is used to transform the 
proofs into another XML (i.e., file GSN-XML in Figure 5) 
logically representing a safety case, by applying the templates 
that have been defined in Section 3. The file format was de- 
signed so that the derived safety cases can be easily be 
adapted to different tools or applications. Finally, to present 
the resulting safety case graphically, we use a Java program 
to layout the logical information which involved some mathe- 
matical calculations in positioning the argument and to con- 
vert it into the standard Adelard ASCE file format. Figure 5 
summarizes the safety case generation process. 


We can thus use two different justifications to mark this dis- 
tinction. Note that this is only a simplification of the presenta- 
tion and does not change the structure of the underlying 
proof, nor the validity of the original goal. It is thus different 
from using a relevant implication [2] under which A => B is 
only valid if the hypothesis A is actually used. 



Figure 6: Hypotheses Handling in =>-Introduction Rule. 

In order to minimize the number of hypotheses tracked by the 
safety case, we need to analyze the proof tree from the leafs 
up, and propagate the used hypotheses towards the root. By 
revealing only the used hypotheses as assumptions, the valid- 
ity of their use in deriving the proof can be checked more 
easily. In our work, we also highlight the use of the external 
certification assumptions that have been formulated in isola- 
tion by the safety engineer. For example in Figure 7, the use 
of the hypothesis has_unit(tptp_float_7_0e_minus_l, 
ang_vel), meaning that a particular floating point variable 
represents an angular velocity, which has been specified as 
external assumption, is tracked properly in the safety case, 
and the validity of its use in deriving the proofs can be 
checked easily. 



Figure 5: Safety Case Generation Process. 


Figure 7: External Hypothesis. 


5 Hypothesis Handling 

An automated prover typically treats the domain theory D and 
the certification assumptions A as premises and tries to derive 

A (D U A) a P => C from an empty set of hypotheses. As the 
proof tree grows, these premises will be turned into hypothe- 
ses, using the =>-introduction rule (see Figure 3). In all other 
rules, the hypotheses are simply inherited from the goal to the 
subgoals. However, not all premises will actually be used as 
hypotheses in the proof, and the safety case should highlight 
those that are actually used. This is particularly important for 
the certification assumptions. We can achieve this by modify- 
ing the template for the =>-introduction (see Figure 6). We 
distinguish between the hypotheses that are actually used in 
the proof of the conclusion (i.e., Ai,...,A k ) and those that are 
vacuously discharged by the =>-introduction (i.e., A k+1 ,..,A n ) 


6 Application to Muscadet 

We illustrate our approach by converting proofs created by 
the AutoCert certification tool [11], which takes a set of re- 
quirements, and a domain theory consisting of logical axioms 
and so-called annotation schemas. These are used to infer 
logical annotations and construct proof tasks which are sent to 
the ATP in order to create proofs that the code complies with 
the requirements. 

For these experiments, we used the Muscadet [19] theorem 
prover during the formal certification of the initialization 
safety of a component of an attitude control system. Muscadet 
is based on natural deduction, but to improve performance, it 
implements a variety of derived rules in addition to the basic 
rules of the calculus. This includes rules for dedicated equal- 






ity handling, as well as rules that the system builds from the 
definitions and lemmas, and that correspond to the application 
of the given definitions and lemmas. Figure 8 shows the re- 
sulting safety case for a proof found by Muscadet. For some 
of the “book-keeping” rules (e.g., the elimination of function 
applications in the elifun-rule in Figure 8) we have not yet 
defined dedicated safety case templates; these rules are repre- 
sented by a generic strategy node. 



Figure 8: ND Proof Safety Case 

7 Proof Abstraction 

Directly converting the Muscadet-proofs into safety cases is 
unfeasible in most practical cases because the proofs contain 
too many elementary and book-keeping steps. It is thus neces- 
sary to abstract the proof. Here, we can apply different ap- 
proaches. For example, we can remove some of the book- 
keeping rules (e.g., return_proof) that are not central to the 
overall argumentation structure. Similarly, we can collapse 
sequences of identical book-keeping rules into a single node. 
In general, however, we try to restructure the resulting proof 
presentation to help in emphasizing the essential proof steps. 
In particular, we can group sub-proofs that apply only axioms 
and lemmas from certain obvious parts of the domain theory 
(e.g., ground arithmetic or partial order reasoning) and repre- 
sent them as a single strategy application. Figure 9 shows an 
example of this. Here, the first abstraction step collapses the 
sequences rooted in G13 and G14, noting the lemmas which 
had been used as strategies as justifications, but keeping the 
branching that is typical for the transitivity. A second step 
then abstracts this away as well. 

8 Related Work 

Other approaches have been used to address concerns with 
using proofs for assurance purposes. Many of them try to 
bring formal proofs into a form closer to human reasoning, to 
aid with their understanding. Proof visualization tools (e.g., 


[22]) present the proof in a graphical form, but quickly get 
overwhelmed by the proof size. Proof verbalization (e.g., [6, 
15]) transforms the proofs into natural language but the ex- 
planations are often too detailed. Proof abstraction groups 
multiple low-level steps that represent recurring argumenta- 
tion patterns into individual abstract steps and thus accentu- 
ates the hierarchical structure of the proof [10] but has so far 
only been applied to interactively constructed proofs. Our 
work combines abstraction, verbalization and visualization to 
reveal and present the proofs underlying argumentation 
structure and top-level assumptions. 

Alternatively, proof checkers [18, 23] have been used to in- 
crease trust in formal proofs, by demonstrating that every 
individual step in the proof is correct. However, proof check- 
ing does not address the real problem: while errors in the im- 
plementations of pro vers do occur, they are very rare [12]; 
errors and inconsistencies in the formalization of the domain 
theory in contrast are much more common, but these are not 
detected by the standard proof checking techniques. 

9 Conclusions 

We have described an approach to derive safety cases from 
software safety proofs found by ATPs. The safety cases serve 
as a traceable argument that shows validity of the proof. They 
also highlight and properly track hypotheses that are actually 
used in deriving the proof, and thus reveal where the proofs 
depend on top-level assumptions. The safety cases provide a 
“structured reading guide” for the safety proofs and make 
clear the interactions underlying the proofs. Hence, assurance 
is no longer implied by the trust in the ATP but follows from 
an explicitly constructed argument for the proofs. 

The work we have described here is still in progress. So far, 
we have automatically derived safety cases for ND proofs 
found by the Muscadet prover [19]. We are currently working 
on safety case templates for the remaining inference rules 
used by Muscadet. We plan to make this technique applicable 
to generally more powerful resolution provers by converting 
the resolution proofs to ND style [14]. The straightforward 
conversion of ND proofs into safety cases is far from satisfac- 
tory as they typically contain too many details. In practice, a 
careful use of abstraction is needed [10] and we are working 
on more abstraction techniques. 

This work complements our previous work [4] where we used 
the high-level structure of annotation inference to explicate 
the top-level structure of such software safety cases. We con- 
sider the safety cases as a first step towards a fully-fledged 
software certificate management system [9] which will pro- 
vide storage and reporting capabilities for all artifacts. We 
also believe that the result of our research will be a compre- 
hensive safety case (i.e., for the program being certified, as 
well as the safety logic and the certification system) that will 
clearly communicate the safety claims, key safety require- 
ments, and evidence required to trust the software. 










Figure 9: Abstraction of Safety Cases 
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